Neural-PIL: Neural Pre-Integrated Lighting for Reflectance Decomposition
The task of decomposing a scene into its underlying physical properties, such as geometry and materials, is useful for applications such as view synthesis, relighting, and object insertion.
Image credit: Piqsels, CC0 Public Domain
A recent paper on arXiv.org aims to recover the 3D shape and spatially varying bidirectional reflectance distribution function (BRDF) of an object imaged under different illumination conditions.
The authors suggest a novel pre-integrated lighting network. It converts the illumination integration process used in rendering into a simple network query. An empirical analysis of synthetic datasets shows that the decomposition network using the novel networks can estimate the more accurate shape and material properties compared to the prior art.
The results of the model can be used to generate high-quality relighting and view-synthesis results with finer details compared to existing approaches.
Decomposing a scene into its shape, reflectance and illumination is a fundamental problem in computer vision and graphics. Neural approaches such as NeRF have achieved remarkable success in view synthesis, but do not explicitly perform decomposition and instead operate exclusively on radiance (the product of reflectance and illumination). Extensions to NeRF, such as NeRD, can perform decomposition but struggle to accurately recover detailed illumination, thereby significantly limiting realism. We propose a novel reflectance decomposition network that can estimate shape, BRDF, and per-image illumination given a set of object images captured under varying illumination. Our key technique is a novel illumination integration network called Neural-PIL that replaces a costly illumination integral operation in the rendering with a simple network query. In addition, we also learn deep low-dimensional priors on BRDF and illumination representations using novel smooth manifold auto-encoders. Our decompositions can result in considerably better BRDF and light estimates enabling more accurate novel view-synthesis and relighting compared to prior art.
